Papers with task-specific modules
OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing task-oriented dialogue systems lack ontology-aware pretraining methods for task-orientated dialogue. |
| Approach: | They propose an ontology-aware pretrained language model (OPAL) for end-to-end task-oriented dialogue (TOD) . they propose to pretrain on large-scale contextual text data to bridge the gap between the pretraining method and downstream tasks. |
| Outcome: | The proposed model achieves an exciting boost and obtains competitive performance even without any TOD data on CamRest676 and MultiWOZ benchmarks. |
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Experimental results show that the proposed model outperforms single-task baseline by 3% and multi-task (without instruction) baseline by 18% on an average. |
| Approach: | They propose a unified model that can learn all 32 instruction tasks of the BoX without any task-specific modules. |
| Outcome: | The proposed model outperforms single-task baseline by 3% and multi-task (without instruction) baseline by 18% on an average. |
Zero-shot Cross-lingual Transfer With Learned Projections Using Unlabeled Target-Language Data (2023.acl-short)
Copied to clipboard
| Challenge: | Zero-shot cross-lingual transfer is enabled by pairing the language adapter in the target language with an appropriate task adapter within a source language. |
| Approach: | They propose to use unlabeled text to enhance zero-shot transfer by pairing language adapters with task adapters in a target language. |
| Outcome: | The proposed framework improves on three cross-lingual tasks with up to 11% relative improvement in Named Entity Recognition (NER), Question Answering (QA) and Natural Language Inference (NLI). |
Analyzing Modular Approaches for Visual Question Decomposition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modular neural networks without additional training have been shown to surpass end-to-end neural networks on challenging vision–language tasks. |
| Approach: | They propose to use BLIP-2-based modular neural networks without additional training to build programs and a number of skill-specific, task-oriented modules to execute them. |
| Outcome: | The proposed methods outperform end-to-end neural networks on vision language tasks and retain performance when they use task-agnostic selections. |
Semi-Supervised Lifelong Language Learning (2022.findings-emnlp)
Copied to clipboard
Yingxiu Zhao, Yinhe Zheng, Bowen Yu, Zhiliang Tian, Dongkyu Lee, Jian Sun, Yongbin Li, Nevin L. Zhang
| Challenge: | Existing methods to learn languages only focus on supervised learning, and unlabeled data is underexplored. |
| Approach: | They propose a semi-supervised lifelong language learning setting where a model learns sequentially arriving language tasks with both labeled and unlabeled data. |
| Outcome: | The proposed model outperforms baseline models on various language tasks and is effective and superior to existing models. |
Hit the Nail on the Head: Parameter-Efficient Multi-task Tuning via Human Language Intervention (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that PEFT on small pre-trained language models improves multitasking capabilities. |
| Approach: | They propose a multi-task learning framework that enables transfer of prior knowledge across tasks . they attach task descriptions to input samples and map them to task embeddings . |
| Outcome: | The proposed method improves performance on a T5 model and in decoder-only models . |
Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Effective cross-lingual transfer is hindered by performance gaps and the scarcity of fine-tuning data in many languages. |
| Approach: | They propose a middle-layer alignment objective integrated into task-specific training to improve cross-lingual transfer across languages. |
| Outcome: | The proposed method improves cross-lingual transfer to lower-resource languages and can be merged with existing modules without full re-training. |